In this Simple Talk opinion piece, Greg Low unpacks how AI models actually filter and weigh training data, why the most common answer isn’t always the best one, and why truth itself can depend on timing and context — not just quality.

A while back, I wrote about my experiences with trying to see if ChatGPT could answer questions as though it was a baseball umpire. It did quite poorly but, curiously, it got hard questions right and easy questions wrong – the complete opposite of what I thought it would do. As such, I’m sure that the problem was what it was learning from.

Furthermore, I realized that the only people who ever discussed the hard concepts were people who knew what they were talking about. So many other people had an opinion – often wrong – about the easy issues.

Naturally, this left me wondering: where exactly do LLMs really learn from?

Learning from the wrong code

I was interviewing a friend for a podcast and he pointed out that the same thing applies to code we write. If I look back at code I wrote years ago, I’d probably cringe. Since then, I’ve learned so much more about what to do right.

If an LLM is trained on all that code, from all those years – that’s pretty scary! It’s little wonder that it’s likely to produce sub-standard code.

The challenge is how it would know the difference – and not just for old code. If an LLM is being trained on code that’s simply out there on the internet, it can easily come across a mix of excellent, terrible and beginner code. Or even simplified code examples for teaching purposes…or techniques that were once correct but no longer are.

The list goes on: what about old Stack Overflow answers that have since been superseded, and/or answers that only work for one production version?

And what about all the code that’s been written to demonstrate what not to do – and code that happens to work but is architecturally awful?

Superficially, these can all look quite similar in training data. So, perhaps it’s not “the LLM has no idea who to learn from.”

Instead, it’s more a case of, “the difficult problem isn’t finding examples. It’s knowing which examples deserve to be treated as authoritative, under which circumstances, and at what point in time.”

Another of my friends recently pointed out to me that “experts don’t often write down what THEY think is obvious”.

Learning without enough context

A related and interesting issue relates to code dialects. With the baseball umpiring questions, I noticed that ChatGPT had mixed softball rules with baseball rules. This led me to be believe that it probably wasn’t given enough information about the differences between the two sports when it was being trained.

A similar thing applies to code. I recently saw this AI-produced code for SQL Server:

This is not valid SQL Server code! I think it would work in Db2, but not most other dialects. In SQL Server, that INNER JOIN should be a CROSS APPLY.

Initially, I thought the AI must have seen some SQL code and decided to learn from it, and then mistakenly decided it was relevant to me. That wasn’t the case. When I asked OpenAI what it thought about the code, it told me it should run.

Upon pointing out that it would not run – and asking the bot why it (incorrectly) thought it would – its answer was fascinating:

“Why I thought it might work: I was pattern-matching from SQL dialects and products where a function call in the right side of a join can behave laterally, and I overgeneralized that to SQL Server.

In other words, I mentally collapsed these two ideas into one: “SQL Server supports joining to TVFs” “therefore this correlated INNER JOIN … ON 1=1 should work” That second step was wrong.

So, the real mistake was not checking the correlated versus uncorrelated distinction before answering. You were right to push back.”

OK then, I suppose. So, every individual piece of the answer looks plausible, but the combination is invalid.

Popularity vs correctness

A common programming pattern might occur millions of times because it’s easy, familiar, copied from tutorials, or has been around for 20 years. Unfortunately, a better technique may be comparatively rare.

So, an LLM’s statistical advantage can become a disadvantage. The most common answer is not necessarily the best answer.

I can imagine:

Software development has spent decades accumulating examples of how people did things, but not necessarily examples of how we should do them.

How are LLMs learning?

I’ve spent some time with people involved in training LLMs. Here’s a summary of what I’ve learned about how they do the training.

Basic statistical filtering

Clearly, a model is trained on a very large collection of text assembled from multiple sources. But they don’t just accept all text. They start with substantial filtering and processing.

Now, some filtering is just mechanical. For example, systems can detect and remove (or at least reduce) spam and SEO-generated pages, duplicated or near-duplicated material, and machine-generated gibberish.

It can also identify and remove pages consisting mostly of navigation or boilerplate, certain unsafe or inappropriate material, and extremely low-quality text. They also estimate of the quality of each item to determine if it should be used for training.

For example, compare:

“Some SQL Server isolation levels use MVCC to allow multiple transactions to access different versions of rows concurrently…”

with:

“SQL SERVER BEST DATABASE!!! CLICK HERE DOWNLOAD FREE DATABASE AMAZING SQL TIPS…”

There are signals that make the second one fairly easy to classify as low-quality, and therefore useless for training.

Quality is not the same as truth

After applying the obvious statistical filtering, there’s still an important limitation: quality is not the same as truth.

A beautifully written article can be completely wrong. And a badly formatted Stack Overflow answer written back in 2011 might still contain the exact obscure technical fact someone needs.

So, filtering cannot simply apply a rule to “keep authoritative-looking text.”

Frequency doesn’t always indicate truth

If a fact like “the capital of Australia is Canberra” occurs repeatedly across thousands of independent pieces of high-quality material, but there are also pages saying “the capital of Australia is Sydney”, how should the filtering be applied?

When training, the system can’t just store these as two database rows and choose between them. So, hopefully there’s an overwhelmingly consistent relationship between Australia and Canberra…but this has a nasty consequence.

As I referenced with the baseball umpiring example, widely-repeated misinformation can also be learned. 20,000 sites that all copied the same incorrect information aren’t really 20,000 independent confirmations of the facts.

Website A: “Feature X was introduced in version 1.6.”

Site B: “Feature X was introduced in version 1.6.”

Site C: “Feature X was introduced in version 1.6.”

That looks like three unique sources. However, if B copied A, and C was generated by an AI whose answer was based on B, that’s effectively one source echoed three times.

So, the people doing training need to also assign greater value to particular kinds of material. Carefully edited books, technical documentation, academic material, high-quality reference works, well-maintained software repositories, and educational material can provide useful training examples. But someone has to assess these.

Time-based truth

If I ask an LLM: “Has DuckDB changed the behavior of feature X in the latest release?”, and it searches the internet, it’s not treating every search result equally.

Of course, you’d hope it’s going to prefer the official DuckDB documentation⁠, release notes, or repository over someone’s two-year-old blog post.

But if I asked “what problems are developers actually having with this DuckDB feature?”, the LLM needs to know that GitHub issues, Stack Overflow discussions, blogs and community discussions might be far more relevant.

Put simply, the correct source of truth can totally depend upon the question being asked.

An official page from the Microsoft site could be excellent for “what does Microsoft say this feature does?”, but is likely to be pretty useless for “what frustrations do developers encounter using this feature?”

None of these processes provide a magical mechanism for determining truth.

Summary

LLMs don’t just have a problem with how much they’ve learned. Once you get past simple scenarios, they have a problem with knowing what they should be learning from.

They can absorb an enormous amount of examples, but those examples vary wildly in quality, age, context, dialect, and intent. Popularity can easily be mistaken for correctness, and individually believable items can be combined into something that is simply wrong.

This doesn’t make LLMs useless as programming assistants (far from it) – but it does mean we should be careful about treating confidence, fluency, or frequency as evidence of expertise.

What do you think? Feel free to leave a comment down below – I’d love to hear from you.